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ABSTRACT 


This research is a continuing analysis of Officer of the Deck (OOD) 
proficiency in the U.S. Navy. Data was analyzed from the Surface Warfare Officers 
School (SWOS) OOD simulator assessments conducted with 66 OODs in Newport, 
Rhode Island, from April 2019 to January 2020. Additionally, self-reported 
demographic and experience survey data was used, which was categorized into 
demographics, operational tempo, overall experience, and the recency of that 
experience. Summarizing the experience data, the author observed over 60% of an 
OOD’s time in the first tour is spent inport and total simulator hours are relatively 
low. The findings suggest that increasing simulator training during non-underway 
periods may be an area for improvement. Estimated multivariate regression 
models were used to determine if OOD performance is correlated with 
experience and recency of that experience. Several key variables are correlated 
with proficiency, but a low sample size prevents a conclusion that these 
relationships are statistically significant. Ultimately, continued research is needed to 
more accurately assess OOD performance factors, analyze long-term policy effects, 
and provide input into future policy making. 
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I. 


INTRODUCTION 


In summer 2017, the United States Naval Surface Force (7th Fleet) suffered two 
major collisions within a short timeframe. The USS Fitzgerald and USS McCain, both 
from 7th Fleet, experienced separate incidents at sea with commercial transport ships 
resulting in severe ship damage and the deaths of 17 Sailors. These collisions sent 
shockwaves through the Navy and resulted in a public call for a complete and thorough 
investigation to review what led to two separate highly capable warships catastrophically 
collide during routine operations at sea. The collisions resulted in various investigation 
reports and policy changes to Surface Warfare Officer (SWO) community training, 
operations and Officer of the Deck assessments. 

A. ASSESSMENT OVERVIEW AND METHODOLOGY 

This research is a continuation of the Officer of the Deck (OOD) assessment pilot 
study analyzed previously by Dr. Jesse Cunha an LT Robb Dearth and the pilot study is 
referred to in the paper for comparison. We analyzed 66 assessments that were conducted 
between April 2019 to January 2020 in seven different assessment periods. In this paper, 
our research is referred to as 2019 assessments and the assessments conducted in 2018 is 
referred to as the pilot study. 

The current OOD proficiency assessments are conducted in Newport, Rhode Island 
by the Surface Warfare Officer School (SWOS). Data collected is in the form of an 
assessment grade sheet, survey questionnaire and a Rules of the Road (RoR) and 
Navigation Seamanship and Shiphandling (NSS) exam scores. We combined all data 
observations from the seven assessment periods and utilized statistical analysis software to 
determine summary data and regression analysis. Our results show the sample is a 
representative distribution of the fleet population in terms of homeport, ship class, ship 
type, and commissioning source. 
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B. ASSESSMENT FINDINGS 


The OODs assessed in our research served onboard their ship while policy changes 
were being implemented, and therefore, have not received the increased training now 
required. However, based on current assessments, we found the assessment overall grade 
performance to be 6% unsatisfactorily, 14% marginal, 23% below average, 32% average, 
12% above average, 12% excellent and 2% exceptional on the 1-7 grading scale (Table 
13). When compared to the pilot study's 3 category grades, we find that the overall 
performance is nearly identical. Although a direct comparison is not possible since the pilot 
study analyzed a different population, it still allows for some trend analysis. 1 Furthermore, 
the RoR exam has a 69.7% pass rate and a 42.4% pass rate for the NSS exam administered 
by SWOS. We observe an operational tempo of 64% (17.8 months of an average 27.8- 
month first tour) either in-port or in the shipyard. While the average underway experience 
as an OOD was 252 hours, only 30 hours occurred in the simulator. With the first tour 
length shifting to a longer 30-month tour, in addition to the increased simulator and 
formalized OOD training, we expect OODs to have more operational experience in the 
future. 

Additionally, we conducted statistical regression analysis to determine the factors 
that are associated with OOD proficiency. We find that if an officer passes the decision¬ 
making subcategory or the performance-under-stress subcategory on the proficiency 
assessment they are more likely to pass the competency assessment. We also see that 
officers above the median experience of OOD hours underway are more likely to have a 
higher assessment score. There is a statistically significant relationship between an officer 
above the median number of straits transits conducted and passing the competency as well 
as the total evolutions (sum of all evolutions) and assessment score. Recency analysis is 
based on having any hours or no recent experience due to the low mean and median values 
of the continuous variables. We see a relationship between having any number of hours as 
OOD, underway or in a simulator, and assessment score as well as any evolutions in the 


1 The pilot study was a fleet spot check of OOD performance with varying timeframes of an officer's 
first tour. The current research is of officers who have completed their first division officer tour. 
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past 90 days. Although there is some correlation, we cannot confidently state the 
significance or magnitude of the relationships due to the high significant errors which are 
driven by the low number of observations. 

C. SUMMARY OF RECOMMENDATIONS 

We propose the following recommendations to improve survey quality are 
proposed: 

1. Improve assessment survey. 

2. Update the assessment grade sheet to minimize subjectivity. 

3. Implement an electronic collection format for Surface Warfare Mariner 
Skills Logbook in addition to the current paper version. 

4. Continue OOD assessment research and expand its use to other SWO 
assessment timeframes. 

5. Increase simulator training, specifically leveraging simulators during in¬ 
port, maintenance and shipyard periods. 

Shifting to an electronic survey format reduces the possibility of errors occurring 
from data collection, while also providing a more efficient method of analysis. A focus 
should be to improve the overall grading scale, while not changing its core measurement 
abilities. In the overall performance, 1-7 grading scale, there is a significant level of 
subjectivity of what differentiates grades like an exceptional versus excellent score. 
Adjusting to a 1-5 grading scale and associating a Commanding Officer's expectations of 
each grade would greatly reduce the subjectivity. For example, a grade of 1-unsatisfactory 
could be associated with an officer the Captain does not trust and would not qualify. A 
grade of 2-below average could be considered probationary and only fit for daytime 
watchstanding, while 3-average and above is an officer the Captain trusts to stand watch at 
all hours. High performing officers who score 4-above average and 5-excellent are those 
that could be selected to train prospective OODs. 


3 



Research of an electronic survey format and database must begin now to coincide 
with the Surface Warfare Mariner Skills Logbook. We understand that many changes 
occurred since the collisions and it will take time for those policy effects to be reflected in 
OOD performance assessments. However, research and development must begin on the 
next phase of the logbook, and we believe that the next phase is the database and software 
development. Creating a database for tracking of watchstanding experience would reflect 
what other programs are currently using, both inside the Navy and in the civilian sector. 

Our final recommendations are to expand this assessment and its observations to 
the other evaluation periods in a SWO career. This will allow trend analysis as well as 
statistical regression analysis with tighter confidence intervals on the factors that affect an 
OOD's performance. Expanding to other assessment periods may also allow for SWOS to 
track progression of a SWO throughout their career. Additionally, based on research 
summary findings, we recommend future research into fleet simulator usage, and 
increasing the use of simulators during inport and shipyard periods. The OOD's assessed 
in this research were Ensigns when the collisions occurred, and when policy changes were 
published. Therefore, the effects of policy changes' are not fully implemented in our 
population group and we expect experience mean values to increase in future research. 
When combined with our work future research can provide a more complete trend analysis, 
giving the Navy metrics on OOD performance trends and the impacts of the increased 
formalized training. 
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II. BACKGROUND 


The chapter describes the Surface Warfare Officer community background, theory, 
and current literature on the use of naval simulation training in the Navy. Additionally, we 
will look at the research on the performance effectiveness of simulation-based training. 

A. SURFACE WARFARE COMMUNITY BACKGROUND 

1. Comprehensive Review of Recent Surface Force Incidents 

On October 26, 2017, U.S. Fleet Forces Command (USFF) published their findings 
of the directed investigation into the contributing factors that led to the separate accidents 
involving the USS Fitzgerald and USS McCain (USFF, 2017). Additionally, USFF 
analyzed past naval incidents to determine a trend of causal factors of concern within the 
surface force. The determined contributing factors included: 

non-compliance with safe navigational practices, incorrect action in 
extremis, substandard proficiency of bridge and Combat Information Center 
(CIC) watchstanders, substandard risk management and planning, 
substandard bridge and CIC coordination, substandard CIC performance, 
inadequate use and understanding of technology, practice of not using AIS, 
substandard use of lookouts, watch bill execution, poor log keeping, 
ineffective shipboard training programs, and inadequate fatigue 
management, (p. 35-39) 

They further categorize the causal factors under the areas of operational safety, 
assessment, fundamentals, teamwork, and culture. It was these areas that USFF 
recommended needed improvements and it specifically addresses Surface Warfare Officer 
training, assessment and culture as areas requiring attention to resolve deficiencies. 

As training is a major focus for the SWO community the USFF findings are 
significant, directly stating, “Surface Warfare Officer candidate training is too dependent 
upon on the job training (OJT)” (USFF, 2017). Additionally, they found a lack of 
assessments conducted on SWOs throughout their careers and specifically a lack of 
standardized training. Each ship has different standards and those standards vary depending 
on which ship an officer is assigned to. USFF specifically states that there is a lack of 
refresher training on Nautical Rules and Bridge Resource Management (BRM) at each 
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career milestone while also lacking an effective assessment. They also determined the 
SWO community lacked tools to track sailor experience, including time spent as a 
watchstander and time spent at sea conducting operations (USFF, 2017). In response to the 
USFF Comprehensive Review, various policy changes were made and implemented to 
address the identified deficiencies, as outlined below. 

2. SWO Career Path Changes 

In June 2018, Commander Naval Surface Force (COMNAVSURFOR) issued a 
directive, “Surface Warfare Officer Career Path and Training Continuum” outlining the 
changes to the SWO community career timelines (Commander Naval Surface Forces 
[COMNAVSURFOR], 2018a). Updated junior officer career timeline and milestones are 
as follows: 

a. Junior Officer Training 

• Ensigns will attend two training courses, Basic Division Officer School 
(BDOC) and OOD Phase I prior to reporting to their ship. Previously only 
BDOC was attended and not OOD Phase I. 

• OOD Phase II to be attended upon completion of first tour or prior to 
fleeting up on the same ship. 

• Advanced Division Officer Course (ADOC) to be attended upon 
successful completion of OOD Phase II. OOD Phase II is new and 
previously only ADOC was attended between ship tours. 

b. Tour Changes 

• First Division Officer tour is extended from 24 months to 30 months. 
Option for single long tour of 48 months instead of two separate tours. 

• Second Division Officer tour reduced from 24 months to 18 months. 

• Second Division Officer tour can no longer be served at afloat staff. 
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• Afloat staff tours will now be conducted by a post-division officer. 

• Afloat Department Head staff tours will now be conducted by post- 
Department Head officer. 

• Time between Department Head and Executive Officer tour will be 
reduced from 5.3 to 4.5 years. 

c. Assessments 

• Ten career assessments have been implemented throughout the SWO 

career path and at career milestones. (COMNAVSURFOR, 2018a) 

Additionally, among COMNAVSURFOR’s career path changes is an emphasis on 
the use of simulation-based training (COMNAVSURFOR, 2018a). Also, the instruction 
specifically states an Ensign will receive more than 100 hours of simulator training as part 
of the OOD Phase I course which is due to be fully functional by 2021. 

d. Mariners Skill Logbook 

To track SWO experience, COMNAVSURFOR established the Mariner Skill 
Fogbook. The logbook is split into three major experience types: watchstanding log, 
special evolution, simulator training (COMNAVSURFOR, 2019). The logbook is a 
significant change for the community with the intent of allowing better tracking of 
experience and proficiency, allows future analysis and builds a professional mindset for 
the community. The goals of the logbook book are accomplished through the tracking of 
the major experience categories by the watchstander, quarterly verifications (Figure 1) and 
an end of tour summary letter of experience (Appendix A). 
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Commanding Officer Quarterly Endorsement Page 

l certify completion of 150 Underway hours, 10 Simulator hours. 

Sea and Anchor Detail, 

Replenishment at Sea, 2 

Anchoring Evolutions, and 1 

Straits Transits during the period 

from 01 Jan 2018 to 31 Mar 2018 . 



_//Signed//_ 

Commanding Officer 

Comments: 

Qualified OOD 15 January 

OOD for Night RAS 10 February 

OOD SOH transit 14 February 

Officer needs to focus on division of labor among bridge watch team 



Figure 1. Mariners Skill Logbook Quarterly Endorsement. 

Source: COMNAVSURFOR (2019). 


3. Proficiency Requirements 

On November 13, 2018, COMNAVSURFOR released his instruction that provides 
direction on minimum watchstander proficiency requirements and requalification 
standards. Requirements by watch station are as follows (Commander Naval Surface 
Forces (Commander Naval Surface Forces [COMNAVSURFOR], 2018b): 

• Officer of the Deck, Tactical Action Officer and Combat Information 
Center Watch Officer - 1 watch every 45 days. 

• Junior Officer of the Deck and Conning Officer - Not applicable. 

• Engineering Officer of the Watch and Combat Systems Officer of the 
Watch- 1 watch every 30 days. (p. 1-3, Enclosure 1) 

Additionally, if proficiency is not maintained watch standing proficiency refresh 
requirements typically involve conducting the watch station under supervision (under 
instruction) of a proficient watchstander or a higher positioned person. 
COMNAVSURFOR approves maintenance of proficiency either underway or in port, 
utilizing a simulator. Results will be logged utilizing the newly implemented Surface 
Warfare Mariners Skills Fog Book, and also tracked by the Senior Watch Officer onboard 
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4. Circadian Rhythm Fatigue Management 

In 2017, Commanders of U.S. Pacific and Atlantic fleet both issued their joint 
instruction COMNAVSURPAC/COMNAVSURFLANT INST 3120.2 Comprehensive 
Fatigue and Endurance Management Policy (Commander Naval Surface Force U.S. Pacific 
Fleet [COMNAVSURPAC], 2017). In the Surface Force, the review addressed fatigue (and 
the culture that surrounds it) as a contributing factor in the 2017 collisions and the culture 
that surrounds it. The instruction directs units to use the circadian rhythm-based routine 
into the unit in order to minimize sub-optimal alertness as a result of fatigue, by 
implementing fixed watches and protected sleep periods. The instruction goes on to specify 
the minimum amount of sleep required: either an uninterrupted 7-hour period or a 5-hour 
period with a 2-hour nap. 

5. Surface Warfare Officer Career Manual 

On December 16, 2019, COMNAVSURFOR issued his newest instruction titled, 
Surface Warfare Officer Career Manual. It cancelled 5 instructions, including 2 instructions 
previously discussed with the intention to create a one-stop instruction for community 
instructions and directives (COMNAVSURFOR, 2019). Its purpose, “to establish the 
single SWO community governing document, providing the requirement and milestones 
from accession through major command” (p. 1). Additionally, it is split into 5 chapters, 
“watchstander proficiency requirements, mariner skills logbook requirements, SWO 
qualification and authority to wear the insignia, SWO milestone mariner skills assessments 
evaluations and competency checks and surface force command requirements” (p. 3-4). 

With a review of the document, there were no major changes and that the full intent 
was as stated in the purpose. From experience, this is welcomed and a significant change 
for career planning and SWO community operations. To cancel 5 instructions and combine 
them into a single document eases the administrative burden on the community and eases 
the task of remaining up to date on the current instruction revision by only having to 
reference the single document vice multiple. 
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B. LITERATURE REVIEW 

1. Naval Simulator Theory 

Since the early 2000s, research has extensively analyzed the current state of the 
Navy’s use of simulators and recommended changes to better utilize simulation-based 
training. For example, RAND National Defense Research Institute conducted a study on 
the use of simulated training by the U.S. Navy in 2005 focused on “assessing the potential 
of reduction or augmenting underway training by completing more training exercises 
through simulation” (Yardley, Thie, Schank, Galegher, & Riposo 2005). Although some 
of the policies have varied since the study was conducted, it provides great insight into the 
analysis conducted on the viability of simulator training. Yardley et al. (2005) determined 
that training representatives for the U.S. Navy were concerned that increasing the use of 
simulators for training would result in a reduction of competency. The preferred training 
method and instilled culture was underway training being the best method of learning for 
SWOs. 

The recommendations from Yardley et al. (2005) for the naval service simulation 
training are summarized as follows: “clearly define the goals of training, specify measures 
of effectiveness for training, utilize multiple approaches to reduce underway training, and 
develop an investment strategy into quality simulators. Additionally, they recommend 
developing a simulation strategy that clarifies and determines what simulators can be used 
for as an equivalent for training and proficiency” (p. 55-62). Yardley et al. (2005) 
ultimately determined that the Navy was conducting most exercises underway, even when 
there was an approved in-port equivalency. They also determined that an underway on the 
job training culture was one of the significant obstacles for expanded simulator use. Based 
on personal experience, this job training culture persists when it comes to Navy fleet best 
training practices. However, that may be shifting as the simulator training initiative has 
been prioritized since the collisions of USS McCain and USS Fitzgerald. A clear and 
focused training strategy and investment into quality simulations is imperative for their 
continued effective adoption. 
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Similarly, in 2002, Schank, Thie, Graff II, Beel, and Sollinger (2002) conducted an 
analysis of the naval services current tactical training and the use of simulators compared 
to our allies' practices and other branches within the Department of Defense (DoD). They 
examined three areas for training: the F/A-18, the P-3C, and the DDG-51 class, in addition 
to other services and allies (Schank et al., 2002). They found that simulator training did not 
have a large role in the F/A-18 training model but was significant in the P-3C and DDG- 
51 training when it came to Anti-Submarine Warfare training. The recommendations in 
Schank et al. (2002) are similar to those of Yardley et al. (2005) in that the Navy needs 
additional measures to gauge readiness and to establish a goal for future live and simulator 
training. Essentially the Navy must analyze simulators, if they are to have a greater return 
on investment than live training. 

2. Simulator Training Effectiveness 

From a different perspective, Weaver (2013) conducted thesis research on student 
examination performance with simulator use during Rules of the Road (RoR) training 
compared to students who only received classroom training. To compare. Weaver analyzed 
27 volunteers who used simulators and 341 participants who only received the classroom 
training. For his research, the control group was the 341 participants whose results were 
provided by the Surface Warfare Officers School and a pre-RoR and post-RoR exam was 
conducted on the treatment/experimental group of 27 between and the pilot study and 
experimental group (Weaver, 2013). 

Weaver’s results showed a statistically significant higher difference in the RoR 
exam scores of the treatment group when compared to the control group of students. He 
showed the pilot study of 6 students mean scores increased from approximately 77% to 
93% and the experiment group of 21, increased from 85% to 98%. In comparison 97.94% 
of students scored below 90% after their courses of instruction in the control group utilized 
(Weaver, 2013). While his results speak to the benefits of simulator training, the estimates 
may be biased because the treatment group were self-selected volunteers who may have 
had a higher aptitude for succeeding in the training. Future research in this field would 
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benefit from the use of randomized controlled trials, instead of voluntary participation of 
the treatment group. 

Additionally, in 2012, Reber and Bernard reviewed the performance of a Surface 
Warfare Officer Instructor course on students when utilizing a simulator-based training as 
opposed to students who did not prior to the use of the Conning Officer Virtual 
Environment (COVE) simulator. From experience, COVE is a specific type of simulator 
that utilizes a headset that the operator wears and uses voice commands to control a ship 
in the simulated environment. They hypothesized that the experimental group would 
perform at a higher level or at an equal level of the control group in various graded 
categories (Reber & Bernard, 2012). Reber and Bernard’s experiment included a total of 
21 participants, with 11 students in the treatment group and 10 students in the control 
group. They concluded that students who used the semi-immersive game prior to 
performing in the SWOS instructor led COVE, performed at a higher and more proficient 
level than the control group. Additionally, they conducted an end of experiment survey and 
the students believed that the game helped them prepare better for the COVE simulator 
exercise. Like Weaver’s ROR research, Reber & Bernard’s treatment group were 
volunteers, which exposes the research to potential self-selection bias and is therefore, a 
consideration when reviewing the results of the COVE assessment. The small sample size 
is also problematic as a stronger experimental model would include a larger sample size 
and randomly selected participants, to minimize the potential for biases. 

Additionally, Tsoukalas, Papachristos, Stefanakou, Tsoumas, and Nikitakos (2014) 
assessed simulator-based training in the Athens Merchant Maritime Academy, intending 
to evaluate the effectiveness of the use of simulation of marine engines. Their study 
surveyed 116 participants (Tsoukalas et al., 2014), and their method was broken into two 
areas; focusing on the course structure and technical aspects for clarity of research. The 
results were that the students’ perceived the courses as adequate and were pleased with the 
simulators for learning (Tsoukalas et al., 2014). The conclusion of the research was that 
overall students are satisfied with the simulator instruction for training but that there are 
areas of improvement in increasing simulator study time and their problem-solving 
teaching design to improve learning efficiencies (Tsoukalas et al., 2014). This research, 
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although focused specifically on the use of simulator training at the Athens Merchant 
Maritime Academy, is applicable to simulator-based training and evaluation methods in all 
areas: a simulator-based training mechanism is only as good as the performance results and 
the perception from the students. However, investment into a simulator training system 
may not be returned. Restructuring is needed, if there is no sign of increase in performance 
and/or is student’s feel the training is inadequate. This same methodology can be applied 
to SWOSs simulator-based training and evaluation, and must be considered when 
reviewing their training curriculum and goals. 

Sonu, Liu, Louis, and Dascalu (2019) conducted another supporting study as part 
of the 2019 International Conference of Information Technology. They specifically 
presented research on the effectiveness of pairing simulator training with learning RoR and 
the way active learning is more effective than passive learning. The authors created a ship 
simulator where they could compare the effects that simulation-based learning would have 
on student learning compared to a traditional method. Their simulator allowed students to 
visually learn RoR and actively control ships’ maneuvering positions. The Sonu et al. 
(2019) study involved 44 students split into equal parts control and experimental group 
where both groups were given pre- and post-tests. Their control group utilized internet RoR 
study material and the experimental, utilized the RoR simulator for self-study. Sonu et al. 
(2019) found that students who utilized RoR simulation-based learning did significantly 
better than students who only used traditional learning methods, with a 20% mean score 
improvement on the post-test. They also found that the students who utilized the simulator 
found it effective and a moderate level of difficulty compared to the control group who 
found the traditional method of learning difficult to learn navigation concepts. 

The Sonu et al. (2019) study shows that use of simulation-based training for 
learning complex material like RoR has a meaningful positive impact on student 
performance. This study supports the notation that the Navy should continue exploring the 
best methods of implementing simulation-based training for learning and proficiency. 
Interestingly, while they found that 70% of students thought the simulation training method 
beneficial, 22% did not, which highlights the fact that every student has a different optimal 
learning style. The authors add to supporting research into the use of simulator’s for 
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seagoing services like the Navy and the Coast Guard when teaching students these complex 
rules and information. Sonu et al. adds to the simulation learning in an area that is 
particularly important to the SWO community, which is learning and applying RoR, while 
other research papers have explored the impact of simulation learning as a general concept. 

Although their study appears to minimize the potential for certain biases to affect 
their findings, a proposal to better improve the study is to utilize a larger sample size for 
future analysis. This will reduce the likelihood that their findings were an abnormality from 
the small sample size. To solve this would be to use a mixed learning method approach of 
both classroom and simulation training combined. Our research will add to their area of 
research and focus on the variables that affect an OOD’s performance and proficiency, 
which directly uses RoR in the simulator scenarios conducted at SWOS. 

3. U.S. Government Accountability Office Report 

Two years following the Navy’s comprehensive review, in November 2019 the 
U.S. Government Accountability Office (GAO) published their report evaluating the 
changes made by the Surface Warfare community. The stated purpose, “(1) describe the 
changes to the Navy has made or planned to make to the SWO ship-driving training since 
the 2017 collisions and (2) assess the extent to which the Navy has taken actions to evaluate 
the effectiveness of those changes” (U.S. Government Accountability Office [GAO], 2019, 
p. 2). GAO determined the effectiveness of changes by conducting interviews, in addition 
to reviewing instructions and administrative changes. GAO also discussed the changes 
supported by an increase in simulator training, specifically “by 2021 plans to triple the 
number of ship-driving training hours when compared with the amount of training SWOs 
were required to receive prior to the collisions” (p. 10). They determined that when the 
Navy officially establishes the four Junior Officer training courses (BDOC, ADOC, OOD 
Phase I and OOD Phase II) the number of hours required will increase from 174 hours to 
535 hours of training (GAO, 2019). 

GAO (2019) has also found the Navy lacks assessment capabilities of policy 
changes that have occurred in the Surface community since the collisions. They state the 
following regarding the Navy's actions: 
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Specifically, and described in detail below, in planning an approach for 
evaluating its efforts, the Navy has not (1) solicited fleet-wide feedback 
on the quality of increased ship-driving training, (2) planned to routinely 
conduct ship driving competency assessments, (3) provided standard 
criteria for qualifying Officer of the Deck candidates, and (4) determined 
how to analyze and use information from logbooks that SWOs are 
required to complete. (GAO, 2019, p. 16) 

Another startling find was that in an OOD assessment spot check conducted 
January to March 2018 of 164 SWOs, 84% had concerns, with 18% reporting significant 
concerns (GAO, 2019). They also found that the required Surface Warfare Mariner Skills 
Logbook lacked any, “specific, measurable plans to analyze and use these data or to assess 
the completeness of these data” (p. 27). GAO (2019) ultimately made four 
recommendations to the Navy summarized as implementing a feedback mechanism from 
the fleet, conducting regular OOD assessments, SWOS establishing standardized OOD 
criteria, and establishing an analysis method for the Mariner Skills Logbook data. 

The GAO report highlights important concerns as a result of the instructions and 
training programs implemented after the collisions. The first step in changing the training 
culture in the community is to establish the new policies, and the next step is to close the 
loop on evaluation of effectiveness and the recommendations GAO proposals are 
appropriately needed for the SWO community to take the next step. A point of concern is 
that the previous research from the early 2000’s states these same concerns with Navy 
simulator training as stated above. If the Navy does not address these concerns and 
questions then there is the possibility that they will go down the wrong path and not use 
resources effectively. Our research is intended to evaluate and improve the OOD 
assessments conducted at SWOS that are used as a type of spot check, as not every student 
conducts the assessments in our dataset. With this research, we can continue to improve 
the data collection methods used to improve the quality of data to allow accurate analysis 
of factors that determine OOD proficiency. 
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4. OOD Pilot Study Assessment 

Cunha and Dearth (2019) published a technical report on research evaluating the 
SWOS OOD Assessment Pilot Study conducted on 164 OODs in 2018. 2 Their research 
intent was to evaluate the data collected, evaluate the quality of data and improve processes 
by SWOS for future data collection. The research results provided the first empirical 
estimates of the relationship between proficiency and knowledge, skills, experience and 
currency (Cunha & Dearth, 2019). Although they did find factors such as experience, skills 
and knowledge can be indicators of proficiency, issues identified in the pilot study data 
resulted in analysis limitations. 

Cunha and Dearth identified limitations on the data collected in 2018 as not having 
a variable that accounts for proficiency directly and some of the time variables were 
collected in bins instead of as a continuous variable. Their primary recommendations to 
improve the quality of future data collection were to add additional variables as well as 
record the variables in a continuous form. They also stressed the importance of the 
Mariners Skills Logbook and how that can improve the quality of data and allow more 
detailed analysis for training program input. Cunha and Dearth (2019) further 
recommended logbook improvement that include adding variables and adding electronic 
tracking format to allow better data collection. 

Importantly, Cunha and Dearth conducted the initial pilot study assessment on 
OOD proficiency, which is a valuable first step for SWOS. By running initial regressions, 
they were able to provide initial proficiency analysis and identify where data collection 
methods require improvement to allow more accurate future analysis. Many of the 
recommendations they proposed in their research were adopted by SWOS for OOD 
Assessments in 2019 and 2020. Our research will analyze the data collected after adopting 
some of their proposed changes to both the assessment grade sheet and the survey data 
collected. The intent is to continue to improve the processes utilized for data collection and 
analysis to provide support to SWOS for future program reviews and help address the 


2 Dearth's master’s thesis (Dearth, 2019) was written before the technical report. 
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criticisms as stated in the GAO 2019 report, Navy Readiness Actions Needed to Evaluate 
Effectiveness of Changes to Surface Warfare Officer Training. 

5. Summary 

For the past two decades, there has been significant research by both the Navy and 
civilians on the use of simulator training and its effectiveness. The common theme of the 
literature is that simulator use is effective if designed properly and if there is a clear 
guidance for its use. The literature shows that the Navy has not implemented simulator- 
based training efficiently into a culture that prefers on the job training at sea. It also shows 
that the Navy lacks infrastructure to analyze simulator performance and assessments and 
requires a focus on improving the quality of data recorded as well improved collection 
methods. The SWO community has made significant changes to training and job policies 
intended to improve the Surface Fleet’s professional abilities as well as ensure 
watchstanders are alert. The next step is to focus on improving the system designed to 
assess a SWO’s abilities throughout their career. 
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III. SWOS OOD ASSESSMENT AND SURVEY 


In this chapter, we will discuss the data and collection methods of the OOD 
assessments and survey conducted by SWOS in Newport, Rhode Island. The assessments 
ongoing are a response by SWOS to the collisions of the USS McCain and USS Fitzgerald 
and the Comprehensive Review conducted by U.S. Fleet Forces. Recommended 
modifications were made following the pilot study by Cunha and Dearth (2019) and are 
utilized in the current assessments. Pilot study and current assessment data and the 
differences are outlined below. 

A. PILOT STUDY DATA 

The previous study by Cunha and Dearth in 2019 was conducted using pilot study 
data. The survey data provided by SWOS was already binned and not a continuous variable 
(Appendix B). J The result of their research was an improved assessment grade sheet and 
an improved survey that expanded the amount of data collected. To improve the quality of 
data, they changed the survey format from a binned collection method to a continuous 
numerical method. The proposed changes from their research are what is used in our 
research for evaluation. The below describes the pilot study data format as compared to the 
data used in our research conducted in 2019 and 2020. 

In the pilot study, the simulator assessment overall performance was evaluated 
using three metrics: completed with no concerns, completed with concerns and significant 
problems. They were also evaluated in five subcategories: management of bridge team, 
bridge resource management, leadership, RoR application and performance under stress. 
Subcategories were evaluated on a 1-4 grading scale (exceeds standards, meets standards, 
requires improvement, unsatisfactory). 

Survey data in the pilot study also collected time variables of months onboard ship, 
months deployed and months to qualify SWO were all collected in a bin format. 
Specifically, the format of collection was a group of month timeframes such as months 


3 Pilot study survey (Appendix B) administered in Cunha and Dearth’s (2019) research. 
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deployed being 0-3 months, 4-6 months, 7-11 months or 12-17 months. They also collected 
information of the Voyage Management System and if they were comfortable and 
confident in using the systems with a binary yes or no answer. 

Evolution experience was collected in the same binned format for CONN and OOD 
experience. Evolution information collected were number of evolutions conducted for pier 
work, underway replenishment approach, anchoring, traffic separation scheme, dense 
traffic operations. Binned metrics on the survey for evolutions experience were none, 1-2, 
3-4, 5-6 and 7 and straits transit were evaluated using a similar format with a higher group 
of binned numbers. They also collected experience underway as CONN and OOD on the 
watch bill based on less than 20 days, 21-99 days, 100-200 or greater than 200 days. 

B. OVERVIEW OF CURRENT ASSESSMENTS 

On an ongoing basis, students were randomly selected using an Excel function and 
the number of assessed students is dependent upon class size and number of available 
assessors. Students are then assigned to an assessor in the order that the Excel function 
randomly selected them. There is no specific or planned pairing of a student to an assessor. 
Prior to formal instruction, on their first day of ADOC, students sign a non-disclosure 
agreement, complete the survey and the examinations in the morning. They are 
subsequently assigned a simulator time in the afternoon for a one-hour period. 

For the assessment, the student is briefed by the Junior Officer of the Deck (JOOD) 
on the scenario utilizing a standard script. They then enter the COVE-3 simulator, meet the 
watch team, and are briefed on bridge, equipment, and equipment settings. The student is 
then asked if they desire to adjust the default settings on the Voyage Management System 
and/or the radar. They are then introduced to the Commanding Officer (assessor) and the 
scenario begins. Each scenario takes approximately 35-45 minutes and is followed by a de¬ 
brief by the assessor. 

OOD assessments were conducted by SWOS staff in Newport, Rhode Island, 
during a span from April 2019 to January 2020. In total, there were seven assessment 
periods with a varying number of students in each. For example, in April 2019, there were 
14 students evaluated and seven students in August. Our research uses four data sources: 
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simulator scenario assessments, Rules of the Road (RoR) exam scores. Naval Seamanship 
and Ship-Handling (NSS) exam scores, and a survey. Simulator assessments are evaluated 
by an 05 or 06 assessor, while the exams and survey are proctored by staff. In total, 69 
students were assessed over the 10 months of our research study. 

1. Officer of the Deck Survey 

A significant piece of information included in the current research is the self- 
conducted 2-page survey that consists of 99 total responses (Appendix C). The major 
changes from the pilot study was the changing of data collection to continuous variables, 
allowing the Officer to write in the actual number. Recency questions were also added to 
the survey to allow analysis for both underway and simulator usage in the past 90 days. 

The survey first collects general demographic data, including name, date, age, rank, 
and gender. The survey then asks about general job-specific information such as years of 
service, commissioning source, college attended, prior enlisted status, and if the student 
received ship-handling training in school on a simulator. Commissioning source allows 
four responses: U.S. Naval Academy, Reserve Officer Training Corps, Officer Candidate 
School and Maritime Academy. These are the four primary methods a SWO will be able 
to earn a commission in the Navy, but are not all inclusive. 4 Prior enlisted information 
consists of two parts, selecting yes or no and the answering what their Military Occupation 
Specialty or rating was if they selected yes to the previous question. 

The next section of the survey is Naval service specific consisting of 13 questions 
that are handwritten or a given set of answers to select. Which ship they served on with an 
example that includes the ship name, class and hull number (i.e., USS Princeton CG-59). 
It continues by asking a series of the following timeline questions: when did they check 
onboard, check-out of their first ship and complete BDOC. Timeline answers are requested 
to be formatted in month and year. It continues by asking when did they qualify OOD, last 
conducted a Naval Seamanship and Ship-Handling Training (NSST) and if they are 


4 Other commissioning sources who were previously allowed to earn their SWO qualification were 
Limited Duty Officers and Chief Warrant Officers but is no longer authorized (COMNAVSURFORINST 
1412.7). 
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qualified SWO with a date of qualification follow on question. As before, the requested 
format is month and year handwritten in. 

The survey then asks about bridge experience in a series of questions designed in a 
table format for the student to fill in with the most accurate information they have available 
from their time onboard. This section starts out with asking if they were in a 3-section 
(OOD, JOOD, CONN) or 4-section (OOD, OOD under instruction (OOD U/I), JOOD, 
CONN) bridge team during normal underway steaming. These are the only choices for 
selection and does not allow a different response, as these are the normal configurations on 
various ship platforms. 

To aid in calculations for the bridge experience tables a note stating, a 4_section 
watch for six months is approximately 1000 hours. This note provides a baseline when 
calculating an estimating the number of hours standing watch. It asks the amount of time 
standing watch as CONN, JOOD, OOD U/I, OOD) in two time periods, total career hours 
and in the past 3 months, and in two locations, on the ship bridge and in a simulator. There 
are 14 sections to fill in as OOD U/I is not asked for the simulator portion. 

The last section of the survey requests information on the number of special 
evolutions that they have conducted on both the bridge and in a simulator in the roles of 
CONN, JOOD and OOD/OOD UI. It is broken into two time periods, total career 
evolutions and number conducted in the past three months, and in two locations, on the 
ship bridge and in a simulator. The section consists of 60 total responses based around the 
number of times they conducted the five following special evolutions: sea and anchor detail 
(entering/exiting port), underway replenishments, anchoring evolutions, straits or Traffic 
Separation Scheme transits and high-traffic density watches. 

2. OOD Proficiency Check 

To assess the OOD simulator proficiency check, the assessor is provided with 
grading guidelines and a grade sheet. 5 Evaluator instructions explain setting up a new tab 


5 The assessment and assessment rubric are not included herein in order to prevent future OODs from 
seeing the assessment before being assessed 
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for each student, utilizing the drop-down menu for grades, or assessing yes and no to 
questions as appropriate. Additionally, the subjective grading consists of a 1 to 5 grading 
scale for subcategories and a 1 to 7 scale for overall assessment categories. Each evaluator 
fills out the appropriate tab for each student as he or she grades simulator performance. 

The assessment grade sheet consists of general information for tracking of data 
(date, name and rank of the student, ship, and evaluator name) and has 9 assessment 
points. 6 The nine assessment points consist of various grading criteria involving a yes and 
no response and an assessed grade from scale 1-5. The assessment skills evaluated in the 
different assessment points are categorized as bridge resource management, internal 
communications, external communications, RoR knowledge, RoR application and ship¬ 
handling execution in the various scenarios. At the conclusion is a summary of the 
individual assessment skills shown in an objective and a subjective summary table. 
Objective summary calculates the number of yes responses by the assessor in the 
assessment scenarios. Subjective summary shows the sum of points accrued in each of the 
assessment skills. Each the objective and subjective table calculates a percentage correct. 

The grade sheet is completed by the assessor evaluating subjective subcategories 
and an overall evaluation. Subjective categories evaluate the skills: management of bridge 
team, formality, presence, and leadership, performance under stress and decision making. 
Subcategories evaluated on a 1-5 scale with the scale evaluated as the following 
performance: 1-unsatisfactorily, 2-below average, 3-average, 4-above average, 5- 
exceptional. Overall performance is evaluated on a 1-7 scale: 1-unsatisfactorily, 2- 
marginal, 3-below average, 4-average, 5-above average, 6-exceptional, 7-excellent. 
Evaluation is graded throughout the assessment with comment sections being provided for 
notes and significant concerns able to be annotated. 

RoR and NSS Exams 

The final metric evaluated by SWOS is student performance on the RoR exam and 
the NSS exam. The RoR exam questions are utilized from the U.S. Coast Guard test bank 


6 The assessment and assessment rubric are not included herein in order to prevent future OODs from 
seeing the assessment before being assessed. 
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and requires a 90% grade or above to pass by normal fleet standards. However, in this 
research because the exam is not incorporated into their ADOC grade, there is no passing 
grade required since it is for research purposes. The exam consists of 40 questions, 20 RoR 
and 20 NSS questions each and they have one hour to complete the exam. 

C. SUGGESTIONS FOR FURTHER IMPROVEMENTS TO DATA 

COUUECTION 

Data quality improvements are always an area of consideration, as lower quality 
data limits the analysis abilities that are able to be conducted. Through the analysis of the 
SWOS collected OOD assessment dataset, areas of improvement have been identified. 
Improving the quality of data collected is a continuous process that began during the pilot 
study assessment and continues with the following recommendations. 

1. Proficiency Assessment Survey 

As stated from Dearth’s (2019) thesis research, a significant improvement is survey 
data collection moving from collection in pre-designated bins to continuous variable 
collection. 7 This allows for an improved quality of data collection and analysis by 
researchers. Now, researchers can bin the continuous data as necessary for evaluation, or 
evaluate the data, in the continuous variable format. 

Currently the survey is conducted in a paper format and then scanned by SWOS 
and sent to NPS researchers. The data is then manually converted from the PDF into excel 
in order to allow analysis utilizing the STATA software. This method is not only 
significantly time consuming for both SWOS and NPS but it also increases the possibility 
of measurement error when transferring the data into excel. Additionally, the completed 
surveys result in readability issues, given all the data is handwritten, and it does not require 
students to accurately and completely provide the requested information. 

Based on our current research and analysis, we recommend that SWOS implement 
an electronic survey format for the following reasons: 


7 Dearth (2019) thesis preceded Cunha and Dearth (2019) technical report. 
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• Minimize measurement error in future research. 

• Electronic storage of data. 

• Ensure students complete survey in proper format. 

• Minimize potential for not completing survey appropriately. 

• Correct readability issues of paper format. 

2. Assessment Grade Sheet Improvements 

From the Cunha and Dearth (2019) pilot study, the assessment metrics were 
expanded and an instruction sheet assists assessors in completing the grade sheet. However, 
the graded metrics are still ambiguous, especially in terms of final overall performance 
grade. An assessor must decide the difference between earning a 6-excellent versus a 7- 
exceptional. This is subjective and the instructions provide no guidance on what deems a 
6 versus a 7. While it may allow greater flexibility for the assessor, it can also result in a 
reduction of data quality. This research recommends a grading scale of 1-5 as used by the 
subcategories, to reduce the ambiguous 1-7 scale currently used to assess overall 
assessment performance. 

The assessment is also prone to varying subjective measures given the different 
assessors and future turnover of the assessors. SWOS attempts to minimize the subjective 
impact with a memorandum containing guidelines for conducting the assessments. The 
differences in grading scales are accounted for by annotating which assessor conducts each 
assessment. In this research, assessors are used as indicator variables. Finally, a 
standardization of evaluation methods is recommended to include improving consistency 
in assessor evaluates, formalization of training on assessment guidelines and to 
standardization of grading criteria. 
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IV. METHODOLOGY AND RESULTS 


In this chapter, we discuss the results of the OOD proficiency assessment and data 
collected by SWOS in Newport, Rhode Island. We discuss the analysis and results of our 
research in two sections, summary of data and regression analyses results. Of the 69 
observations collected between April 2019 to January 2020, three were removed due to 
data quality issues. 8 With the low number of observations, finding statistical significance 
is difficult and for future research, we recommend increasing the study size. 

Summary data below shows that the randomly selected sample looks similar to the 
population. The data used is unpublished and not retrievable by the public and was 
provided by SWOS. We converted survey and assessment grading into a fully electronic 
format for analysis. Following summary data, we show our regression analysis which 
estimates the correlates of OOD performance. 

A. SUMMARY OF DATA 

Table 1 contains the summary statistics of demographic data collected in the survey. 
Survey data was collected as continuous variables and binned where useful for analysis. 
As stated earlier, this is in contrast to Cunha and Dearth’s (2019) research, where SWOS 
collected survey data in bins and required the conversion to an averaged continuous 
variable by the researchers. 

Commissioning source consisted of 20% Officers receiving their commission from 
Officer Candidate School (OCS), 48% commissioning through the Reserve Officer 
Training Corps (ROTC), 30% commissioning through the U.S. Naval Academy (USNA) 
and 2% from the U.S. Merchant Marine Academy (USMMA). The one Officer that 
attended the USMMA was added to the USNA observation for regression analysis. 


8 The first observation was removed because the number of hours standing OOD, JOOD and CONN 
were unrealistic and likely an error. The second observation was removed because the student failed to 
answer a significant amount of the experience survey questions. Third observation was removed due to 
answers being inconsistent and not complete. 
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Ship class include an officer distribution of 15% from Guided-Missile Cruisers 
(CG), 48% from Guided-Missile Destroyers (DDG), 12% from Landing Helicopter Dock 
(LHD), 3% from Landing Helicopter Assault (LHA), 8% from Landing Platform Dock 
(LPD), 2% from Amphibious Command Ship (LCC), and 12% from Landing Ship Dock 
(LSD). The current allocation of Officers in the study represents a wide distribution of ship 
classes that first-tour Division Officers serve onboard with the exception of the Mine 
Counter Measure (MCM) platform, which is not represented. Other classes not represented 
since second-tour Division Officer are typically assigned to their billets is the Littoral 
Combat Ship (LCS), Nuclear Aircraft Carrier (CVN) and Patrol Craft (PC). 

A new variable bin, ship types, was created in this research to collect ships that are 
Cruiser and Destroyers (CRUDES) and Amphibious (AMPHIB). The surface Navy 
community typically refers to officers in these two fields of ship types. Ship type 
distribution is 64% serving on CRUDES and 36% on an AMPHIB. LCC, although not the 
typical amphibious type ship, was included in the AMPHI B bin. 

Homeport was represented throughout the nine major ports that first tour division 
Officers normally are stationed. Homeport distribution is 3% in Everett, 18% Mayport, 
30% Norfolk, three Pearl Harbor, 28% San Diego, 3% Bahrain, 1% Rota (Spain), 6% 
Sasebo (Japan) and 8% Yokosuka (Japan). Homeport was also binned into West Coast (22 
Officers or 33%), East Coast (32 Officers or 48%) and Overseas (12 Officers or 18%) due 
to the low number of observations in some of the individual homeports. Gender is presented 
as 39 male and 27 females, and age is represented and placed in a bin for display 
summarization in Table 1. 
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Table 1. SWOS OOD Proficiency Assessment Demographic Summary 



Percentage 

Observations 

Commissioning source 

Officer Candidate School 

20% 

13 

ROTC 

48% 

32 

US Naval Academy 

30% 

20 

US Merchant Marine Academy 

2% 

1 

Ship class 

Cruiser 

15% 

10 

Destroyer 

48% 

32 

Landing Helicopter Dock (LHD) 

12% 

8 

Landing Helicopter Assault (LHA; 

3% 

2 

Landing Platform Dock (LPD) 

8% 

5 

Amphibious Command Ship (LCC 

2% 

1 

Landing Ship Dock (LSD) 

12% 

8 

Ship type 

Amphibious ship 

36% 

24 

Crusiser or Destroyer 

64% 

42 

Home port 

Everett, WA 

3% 

2 

Mayport, FL 

18% 

12 

Norfolk, VA 

30% 

20 

Pearl Harbor, HI 

5% 

3 

San Diego, CA 

26% 

17 

Manama, Bahrain 

3% 

2 

Rota, Spain 

2% 

1 

Sasebo, Japan 

6% 

4 

Yokosuka, Japan 

8% 

5 

Home port location 

West Coast U.S. 

33% 

22 

East Coast U.S. 

48% 

32 

Overseas 

18% 

12 

Gender 

Male 

59% 

39 

Female 

41% 

27 

Age (years) 

Less than 25 

36% 

24 

25 through 29 

56% 

37 

Greater than 29 

8% 

5 

Observations 


66 


Note: OOD proficiency check assessment data of first tour OODs, collected at SWOS from April 
2019 to January 2020. Unpublished data used for analysis provided by SWOS. Source: (E. Cotter, 
personal communication, January 13, 2020). 
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Table 2 summarized the survey career data. Time variable information pertaining 
to the Officer’s experience includes months onboard first ship, months underway (not 
deployed), months underway (deployed) and months in shipyard. Qualification and 
training time variables are also collected in a month and year format and then converted to 
number of months for each. Total overall experience is recorded by documenting the 
number of hours the Officer stood in various bridge watch stations both underway and in a 
simulator. 

Table 2 displays presents the mean, standard deviation, median, 5% level and 95% 
level of the selected variables. The average Officer spent 27 months onboard their first 
ship, just over 5 months deployed underway and just 5 months underway undeployed. Of 
note, the average time an Officer spent in the shipyard was 11 months. As expected, an 
Officer spends more time standing CONN watch, then JOOD and finally OOD based upon 
qualification progression and pre-requisites. Simulator use is at a much lower level for 
bridge watch stations and special evolution experience. Based on the average time spent in 
a shipyard, simulator usage for training and professional development is an area that further 
research may benefit. 

On average, a significant amount of time was spent in a shipyard, when the Officer 
would not be able to train and maintain proficiency as an OOD or CONN. This is an area 
where significant investment in simulator training can be leveraged for qualification, 
training and proficiency. 
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Table 2. OOD Assessment Total Underway and Simulator Experience. 



Mean 

s.d. 

Median 

5th percentile 

95th percentile 

Months in position 

aboard ship 

27.8 

6.2 

27 

18 

34 

underway, not deployed 

5.3 

4.6 

5 

0 

12 

deployed 

5.0 

3.2 

6 

0 

8 

in-port, not underway 

6.3 

4.8 

5 

0 

15 

in shipyard 

11.5 

7.2 

10 

2 

24 

Months since attending BRM 

10.8 

7.0 

10.5 

1 

24 

Months taken to qualify OOD 

19.5 

6.6 

19.5 

10 

30 

Months taken to qualify SWO 

22.9 

6.6 

23 

14 

33 

Underway hours in 1st tour 

as CONN 

514.7 

388.1 

450 

25 

1200 

as JOOD 

342.0 

257.0 

300 

15 

833 

as OOD UI 

135.5 

167.6 

100 

0 

500 

as OOD 

252.8 

255.0 

180 

5 

850 

in any position 

1245.9 

653.6 

1212 

383 

2318 

Simulator hours in 1st tour 

as CONN 

60.4 

163.6 

20 

0 

250 

as JOOD 

44.6 

139.4 

5.5 

0 

200 

as OOD 

30.0 

52.1 

10 

0 

192 

in any position 

135.0 

332.2 

40 

6 

400 

Overall hours (underway and simulator) 

as CONN 

575.1 

450.1 

477.5 

103 

1540 

as JOOD 

386.5 

337.7 

300 

21 

1100 

as OOD 

283.8 

252.8 

200 

18 

855 

in any position 

1380.9 

816.3 

1340 

414 

3319 


Note: All variables collected as continuous variables. 


Figure 2 displays the distribution of operational tempo recorded in months for 
OODs in this research. Operational experience is shown as months spent underway (not 
deployed), months spent underway (deployed), months inport (not underway) and months 
spent in shipyard. 
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Figure 2. Operational Tempo of First Tour OODs. 


Figure 3 illustrates the qualification timeframe for an officer attaining their Officer 
of the Deck letter and their Surface Warfare Officer designation. An Officer of the Deck 
letter consists of multiple pre-requisite qualifications and the final major pre-requisite 
qualification before earning the SWO qualification/designation. OOD and SWO 
qualification timelines vary based on ship schedule, number of ensigns onboard also 
seeking their qualification, CO training philosophy and other factors. This explains the 
wide variation from under 12 months to above 20 months for qualification in each category. 
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Figure 3. OOD and SWO Qualification Timeframes. 


Figure 4 shows the distribution of underway hours experience as CONN, JOOD, 
OOD and OOD UI. The figure shows a left skewed distribution of experience. 
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Figure 4. Overall Underway Hours Experience. 
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Figure 5 illustrates the distribution of OOD experience underway or using a 
simulator from information provided in Table 2. Overall OOD experience, whether 
underway or in a simulator is skewed left as well with a significant percentage of OODs 
with less than 100 hours of experience. 



Overall Underway and Simulator OOD Hours 

Figure 5. Overall OOD Hours, Underway or Simulator 

Table 3 displays special evolution experience of five major evolutions conducted 
by a Surface Warfare Officer on the bridge: pier work, underway replenishments, 
anchoring evolutions, Traffic Separation Schemes (TSS) and strait transits, and high traffic 
density evolutions. The average number of total underway evolutions conducted by and 
OOD is 25, 31 evolutions as JOOD and 34 as CONN. The middle 50% (median) for some 
evolutions is low compared to the average (mean) value. The average number of evolutions 
conducted is increased because of the Officers that had a higher operational tempo and 
conducting a significant amount of evolutions as shown in the 95th percentile column. 
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Table 3. Total Overall Underway Special Evolutions. 



Mean 


Median 

5th percentile 

95th percentile 

Special Evolutions Underway 

Pierwork 

as CONN 

7.9 

9.7 

5 

1 

15 

as JOOD 

9.5 

31.1 

5 

0 

20 

as OOD 

7.5 

20.0 

4 

0 

18 

Underway Replenishment 

as CONN 

8.8 

8.3 

6 

2 

30 

as JOOD 

4.6 

7.7 

3 

0 

10 

as OOD 

3.6 

7.6 

1.5 

0 

15 

Anchorings 

as CONN 

1.2 

1.7 

1 

0 

5 

as JOOD 

1.4 

3.5 

1 

0 

4 

as OOD 

1.3 

2.4 

0 

0 

4 

TSS and Strait Transits 

as CONN 

4.4 

5.7 

2 

0 

16 

as JOOD 

5.6 

12.8 

2 

0 

18 

as OOD 

4.2 

10.4 

1 

0 

15 

High traffic situations 

as CONN 

12.3 

15.7 

5 

0 

50 

as JOOD 

9.9 

16.1 

5 

0 

30 

as OOD 

9.0 

15.2 

3 

0 

43 


Note: All variables collected as continuous variables. 


Table 4 shows the same variables as depicted in Table 3 but for simulator usage 
instead of underway execution of the evolutions. Simulator mean number of evolutions 
conducted by OOD, JOOD and CONN are overall lower than evolutions conducted 
underway. 
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Table 4. Total Overall Simulator Special Evolutions 



Mean 

s.d. 

Median 

5th percentile 

95th percentile 

Special simulator evolutions 

Pierwork 

as CONN 

6.0 

6.8 

4 

0 

22 

as JOOD 

3.2 

6.2 

1 

0 

12 

as OOD 

3.0 

4.1 

2 

0 

10 

Underway Replenishment 

as CONN 

3.9 

5.0 

2.5 

0 

15 

as JOOD 

1.5 

3.1 

0 

0 

8 

as OOD 

2.1 

3.6 

1 

0 

10 

Anchorings 

as CONN 

1.2 

2.1 

0 

0 

5 

as JOOD 

0.5 

0.9 

0 

0 

2 

as OOD 

0.9 

1.9 

0 

0 

3 

TSS and Strait Transits 

as CONN 

4.6 

6.7 

2 

0 

24 

as JOOD 

2.4 

3.9 

1 

0 

10 

as OOD 

2.7 

3.8 

1 

0 

10 

High traffic situations 

as CONN 

3.8 

3.8 

3 

0 

10 

as JOOD 

2.8 

4.7 

0.5 

0 

10 

as OOD 

2.9 

3.6 

2 

0 

10 


Note: All variables collected as continuous variables. 


Table 5 depicts the total evolution experience as a CONN, JOOD and OOD 
underway or in a simulator and the combined total experience both underway and in a 
simulator. 
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Table 5. Total Evolution Experience, Underway and Simulator 



Mean 

s.d. 

Median 

5th percentile 

95th percentile 

Total underway evolutions 

as CONN 

34.6 

26.9 

25 

8 

81 

as JOOD 

31.0 

66.2 

16 

1 

71 

as OOD 

25.6 

42.4 

12 

2 

88 

Total simulator evolutions 

as CONN 

19.5 

19.3 

14 

2 

71 

as JOOD 

10.4 

16.0 

3.5 

0 

48 

as OOD 

11.7 

14.7 

7 

0 

39 

Total underway or simulator evolutions 

as CONN 

54.0 

34.2 

45.5 

13 

112 

as JOOD 

41.4 

72.6 

20.5 

7 

95 

as OOD 

37.3 

48.1 

22.0 

4 

97 

in any position 

132.2 

133.6 

98 

33 

282 


Note: All variables collected as continuous variables. 


Table 6 displays the experience recency of a Surface Warfare Officer based on self- 
reported information by the Officer. Recency is defined as standing the watchstation or 
conducting the special evolution within the past 90 days. The Officer completes the survey 
on first day of ADOC which minimizes the degradation of the recency variables due to 
time spent in school taking away significance of the variables. 

This research is the first collecting recency experience data, therefore there is no 
baseline of comparison. As shown in Table 6, the average time spent underway as an OOD 
is 41 hours, while the CONN average is three hours. Simulator use of the same watches is 
two hours as OOD and underway one hours as CONN. Special evolutions both underway 
and in a simulator are mostly less than one, on average, being conducted in the past 90 
days. Although the mean is above zero on many recency variables, the median is 
overwhelmingly zero for most variables. 

This is an important finding, showing that the top 95% is pulling up the mean from 
zero when in reality 50% conducted minimal watch or special evolutions as OOD or 
CONN. This is an area that further research would benefit in determining if and how 
proficiency is being maintained based on the self-reported data. 
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Table 6. Underway and Simulator Recency Experience—Within 90 Days 



Mean 

s.d. 

Median 

5th percentile 95th percentile 

Underway hours in past 90 days 

as CONN 

3.1 

16.3 

0 

0 

130 

as JOOD 

14.6 

49.2 

0 

0 

100 

as OOD UI 

2.7 

13.8 

0 

0 

12 

as OOD 

41.8 

96.5 

0 

0 

264 

in any position in past 90 days 

62.29 

109.33 

0 

0 

330 

Simulator hours in past 90 days 

as CONN 

0.7 

3.4 

0 

0 

2 

as JOOD 

0.2 

1.2 

0 

0 

0 

as OOD 

2.1 

6.2 

0 

0 

10 

in any position in past 90 days 

2.95 

8.41 

0 

0 

18 

Overall hours (underway and simulator) 
in past 90 days 

as CONN 

3.8 

17.0 

0 

0 

10 

as JOOD 

14.8 

49.2 

0 

0 

100 

as OOD 

43.9 

95.9 

1 

0 

264 

in any position 

65.2 

108.8 

3.5 

0 

338 


Note: All variables collected as continuous variables. 


Figure 6 shows the distribution of experience for underway hours or simulator 
hours and a combined overall experience in the past 90 days. Recency experience is broken 
down by the watchstanders CONN, JOOD, OOD and OOD UI. The figure illustrates the 
extreme left skewed distribution of recency experience by watchstation. 
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Figure 6. Underway and Simulator Recency Experience— 

Within 90 Days 


Table 7 depicts the number of special evolutions conducted within the past 90 days. 
The mean value of each evolution conducted by the three watchstations was below one 
with the exception of high traffic situations as an OOD, just over one evolution on average 
in the past 90 days. 
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Table 7. Underway Special Evolutions Recency—Within 90 Days 



Mean 

s.d. 

Median 

5th percentile 95th percentile 

Special underway evolutions in past 90 

Pierwork 

as CONN 

0.3 

0.7 

0 

0 

2 

as JOOD 

0.1 

0.4 

0 

0 

1 

as OOD 

0.6 

1.3 

0 

0 

4 

Underway Replenishment 

as CONN 

0.3 

0.8 

0 

0 

2 

as JOOD 

0.1 

0.4 

0 

0 

0 

as OOD 

0.2 

0.6 

0 

0 

1 

Anchorings 

as CONN 

0.0 

0.2 

0 

0 

0 

as JOOD 

0.0 

0.2 

0 

0 

0 

as OOD 

0.3 

0.8 

0 

0 

2 

TSS and Strait Transits 

as CONN 

0.1 

0.5 

0 

0 

0 

as JOOD 

0.0 

0.2 

0 

0 

0 

as OOD 

0.5 

2.0 

0 

0 

2 

High traffic situations 

as CONN 

0.7 

4.9 

0 

0 

0 

as JOOD 

0.1 

0.5 

0 

0 

0 

as OOD 

1.7 

7.5 

0 

0 

5 


Note: All variables collected as continuous variables. 


Table 8 contains the number of evolutions conducted using a simulator instead of 
underway in the past 90 days. All averages are less than one for each recency variable with 
the top 95% only have conducted a few evolutions in the past three months in a simulator. 
Of note, the self-reported data suggests that simulators may not be used frequently in the 
fleet. Future research into fleet simulator usage may be beneficial as the data currently 
suggests this an opportunity that can continue to be improved upon. 
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Table 8. Simulator Special Evolutions Recency—Within 90 Days 



Mean 


Median 

5th percentile 95 th percentile 

Special simulator evolutions in past 90 days 

Pierwork 

as CONN 

0.1 

0.6 

0 

0 

0 

as JOOD 

0.0 

0.0 

0 

0 

0 

as OOD 

0.2 

0.7 

0 

0 

2 

Underway Replenishment 

as CONN 

0.1 

0.6 

0 

0 

0 

as JOOD 

0.0 

0.0 

0 

0 

0 

as OOD 

0.1 

0.5 

0 

0 

0 

Anchorings 

as CONN 

0.0 

0.3 

0 

0 

0 

as JOOD 

0.0 

0.0 

0 

0 

0 

as OOD 

0.1 

0.4 

0 

0 

0 

TSS and Strait Transits 

as CONN 

0.1 

0.7 

0 

0 

1 

as JOOD 

0.0 

0.3 

0 

0 

0 

as OOD 

0.3 

0.8 

0 

0 

2 

High traffic situations 

as CONN 

0.1 

0.6 

0 

0 

0 

as JOOD 

0.0 

0.3 

0 

0 

0 

as OOD 

0.4 

1.0 

0 

0 

3 


Note: All variables collected as continuous variables. 


Table 9 depicts the total recency experience of an Officer underway or in a 
simulator and total recency underway and in a simulator. Findings show the average 
number of evolutions conducted in the past 90 days was four. This finding, although 
informative, does not reflect if the Officer was onboard their ship in the past 90 days. It 
may be possible they were attending another course in addition to leave and transit time. 
The data does not provide those details. 
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Table 9. Total Recency Experience, Underway and Simulator. 



Mean 

s.d. 

Median 

5th percentile 95th percentile 

Total underway evolutions in past 90 days 

as CONN 

1.3 

6.0 

0 

0 

4 

as JOOD 

0.3 

1.3 

0 

0 

2 

as OOD 

3.2 

9.6 

0 

0 

18 

Total simulator evolutions in past 90 days 

as CONN 

0.5 

2.1 

0 

0 

3 

as JOOD 

0.1 

0.5 

0 

0 

0 

as OOD 

1.0 

2.7 

0 

0 

5 

Total underway or simulator evolutions 

as CONN 

1.8 

6.9 

0 

0 

6 

as JOOD 

0.4 

1.4 

0 

0 

2 

as OOD 

4.2 

9.8 

1 

0 

18 

in any position 

6.4 

16.1 

1 

0 

19 


Note: All variables collected as continuous variables. 


Table 10 contains the assessment performance grades of the four subcategories that 
are evaluated in addition to overall performance. Of note, 43.9% of Officers received a 
grade of below average or unsatisfactory on their performance under stress evaluation. This 
is a concerning finding but the intent of the SWO training changes implemented is that 
these scores will improve over time. These below average scores may be indicative of a 
lack of familiarity or not being comfortable performing OOD functions. 


Table 10. OOD Proficiency Assessment Subcategory Performance. 


Assessment Subcategories 

Unsatisfactory 

Below 

Average 

Average 

Above 

Average 

Excellent 

Management of bridge team 

7.6% 

22.7% 

37.9% 

25.8% 

6.0% 

Formality / Presence / Leadership 

7.6% 

24.2% 

39.4% 

24.2% 

4.5% 

Decision Making 

10.6% 

28.8% 

25.8% 

30.3% 

4.5% 

Performance under stress 

9.1% 

34.8% 

24.2% 

22.7% 

7.6% 


Note: Subcategory grading based on a 1-5 scale. 
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Table 11 shows the other assessment metrics, RoR and NSS exams scores. RoR’s 
mean score was a 90%; however, only 69.7% passed the RoR portion of the exam. Passing 
score for RoR is defined as receiving a 90% or higher. In comparison, only 42.4% passed 
the NSS portion of the exam. NSS passing score is defined as an 80% or higher. 


Table 11. RoR and NSS Exam Performance 



Mean 

s.d. 

min 

max 

% pass 

Rules of the Road test 

0.90 

0.08 

0.70 

1 

69.7% 

NSS test 

0.75 

0.12 

0.55 

0.95 

42.4% 


Note: RoR and NSS exam are 20 questions each. 

Figure 7 depicts the distribution of RoR and NSS exam scores as shown in Table 
11. The figure shows the distribution of both exam grades and depicts the failing score 
distribution for each. The passing percentage is concerning for both exams considering that 
all officers in this research have completed their first tour on ship. 




RoR Scores NSS Scores 

Figure 7. RoR and NSS Exam Grade Distribution 
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Table 12 compares RoR and NSS exam performance from 2018 to 2019. 


Table 12. Comparison of 2018 and 2019 OOD Proficiency Assessment 

Performance. 


2018 

Mean 

s.d. 

min 

max 

% pass 

Rules of the Road test 

0.91 

0.08 

0.60 

1 

72.0% 

NSS test 

0.80 

0.10 

0.50 

1 

65.2% 


2019 

Mean 

s.d. 

min 

max 

% pass 

Rules of the Road test 

0.90 

0.08 

0.70 

1 

69.7% 

NSS test 

0.75 

0.12 

0.55 

0.95 

42.4% 


Note: Passing RoR and NSS scores defined as 90% and 80%, respectively. 2018 scores from 
previous research by Dr. Cunha and LT Dearth. Source: Cunha and Dearth (2019). The 2019 scores 
are from SWOS OOD proficiency checks, April 2019-January 2020. 


Table 13 shows the overall performance of Officers who participated in this 
research by conducting the OOD Proficiency Assessment from April 2019-January 2020. 
Of the 66 Officers, 28 Officers (42%) received a below average, marginal or an overall 
grade of unsatisfactory. 21 Officers (32%) received an average grade and only 17 (26%) 
Officers received grade in the above average categories. A concerning finding, 42% of 
Officers assessed scored below the assessment average grade and four failed the 
assessment entirely. 9 


9 The 42% that scored below the average grade received a grade of below average, marginal or 
unsatisfactorily. 
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Table 13. Overall OOD Proficiency Assessment Performance. 


Percentile Observations 


Overall performance 


Exceptional 

2% 

1 

Excellent 

12% 

8 

Above Average 

12% 

8 

Average 

32% 

21 

Below Average 

23% 

15 

Marginal 

14% 

9 

Unsatisfactory 

6% 

4 

Total Observations 


66 


In Table 14, we show a comparison of OOD proficiency performance from those 
assessed in 2018 and the current research in 2019. 10 The underlying data is not a direct 
comparison as the grading collection and grading criteria are different in each year. To 
allow comparison we binned our 7-point overall grade scale into three comparable bins. 

The results are virtually identical with about 66% scoring in the middle categories 
and smaller percentages either showing significant problems or having an outstanding 
performance. This is perhaps not surprising since the changes implemented after the 2017 
collisions would not have immediately pertained to these Officers. The OODs in this 
research are at the end of their first tour, while the pilot study data point were varying 
timeframes. 


10 Current research is referred to as taking place in 2019 and includes an assessment from January 
2020 . 
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Table 14. Comparison of 2018 and 2019 OOD Proficiency Assessment 

Performance. 



% Significant 

% Complete with 

% Complete no 


concerns 

concerns 

concerns 

2018 Overall Performance 

18% 

66% 

16% 


% Unsatisfactory 
and % Marginal 

% Average, 
Below and Above 
Average 

% Exceptional 
and % Excellent 

2019 Overall Performance 20% 

67% 

14% 


Note: 2018 overall performance provided from previous research by Dr. Cunha and LT Dearth. 
Source: Cunha and Dearth (2019). The 2019 performance information is from SWOS OOD 
proficiency checks, April 2019-January 2020. 


B. REGRESSION ANALYSIS 

In this section we will present the findings of our multi-variate regression analysis 
to determine factors effecting OOD proficiency. As part of our analysis, due to degrees of 
freedom considerations, we split the regressions into five separate tables. The tables consist 
of demographics, experience, special evolutions, recency of experience, and recency of 
special evolutions. Recency is defined as performing the watchstation or conducting the 
special evolution in the past 90 days. 

For our regression analysis we combined JOOD, OOD UI and OOD into one 
variable called OOD. This was due to the fact that all three watchstations accomplish a 
similar objective: exposing an officer to the same type of experiences and decision making. 
JOOD is commonly an OOD in training, and in many cases will be recommending the 
decisions and actions to the OOD. JOOD's will also commonly make reports to the ship 
Commanding Officer (CO), further earning their trust and confidence. We analyze two 
separate outcome variables, passing the OOD competency assessment and overall grade on 
the 1-7 scale. The results show that relationships of meaningful magnitude do exist, 
however, due to the low number of observations, we also observe high standard errors. 
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This means that although there are correlations to the outcome variables, we cannot 
confidently state the magnitude of effect of the relationship. 

1. Naval Background and Demographic Analysis 

Table 15 shows the effect of an officer's naval background and demographic impact 
on the probability of passing the competency exam and their overall assessment score (1- 
7 scale). We created binary variables indicating passing each of the four assessment 
subcategory grades: management of the bridge, leadership, performance under stress, and 
decision making. The assessor variables are included in each regression as fixed effects. 

In analyzing the results of Table 5, Column 1 shows a positive relationship between 
passing each subcategory and passing the OOD competency assessment. Additionally, 
column 4 shows correlation between the subcategories and overall assessment grade. We 
see that performance under stress and decision making are highly correlated to both 
outcome variables, with decision making having the highest positive impact on the 
outcomes. This may be because bridge management and leadership are traits are more 
easily learned, but performance under stress and decision making are more innate abilities. 
From experience, these abilities are what CO looks for when qualifying and trusting an 
OOD. This would then mean the assessors would also naturally look for these qualities as 
well. The assessor variables show a wide variation of grading and their effect on the 
outcome variables, which points to the subjective nature of the grading, and the need to 
standardize the grading criteria. Finally, we also see that location variables are not telling 
us much about the outcome variables. 
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Table 15. Assessment subcategory grading correlation of the outcome 

variables 



Pass competency (score>=3) 

Assessment score (scale of 1-7) 


(1) 

(2) 

(3) 

(4) 

(5) 

(6) 

Subcategory performance 







Passed management of bridge category 

0.117 



0.437 




(0.108) 



(0.320) 



Passed leadership category 

0.219** 



0.391 




(0.098) 



(0.290) 



Passed performance under stress category 

0.227*** 



1.055*** 




(0.080) 



(0.239) 



Passed decision making category 

0.514*** 



1.146*** 




(0.090) 



(0.269) 



Platform 







Amphibious ship 


-0.08 

-0.041 


-0.31 

-0.192 



(0.15) 

(0.143) 


(0.41) 

(0.406) 

Commissioning source 







U.S. Naval Academy 


-0.17 

-0.170 


-0.13 

-0.130 



(0.20) 

(0.199) 


(0.58) 

(0.565) 

Naval ROTC 


-0.11 

-0.073 


-0.30 

-0.205 



(0.19) 

(0.187) 


(0.54) 

(0.532) 

Home port location 







Forward deployed 


-0.12 

-0.048 


-0.51 

-0.330 



(0.20) 

(0.199) 


(0.57) 

(0.565) 

West coast 


0.10 

0.121 


0.16 

0.212 



(0.17) 

(0.163) 


(0.47) 

(0.464) 

Exam performance 







Rules of the Road exam 



0.633 



0.858 




(0.939) 



(2.668) 

Navigation, Seamanship & Ship handling exam 



1.187* 



3.585** 




(0.615) 



(1.748) 

OOD assessor 







Assessor 1 

-0.084 

-0.53* 

-0.451 

-0.006 

-1.12 

-0.907 


(0.129) 

(0.30) 

(0.294) 

(0.383) 

(0.85) 

(0.837) 

Assessor 2 

-0.056 

-0.25 

-0.119 

-0.050 

-0.59 

-0.245 


(0.108) 

(0.23) 

(0.234) 

(0.321) 

(0.66) 

(0.665) 

Assessor 3 

-0.029 

-0.24 

-0.048 

-0.134 

-0.76 

-0.242 


(0.093) 

(0.20) 

(0.215) 

(0.276) 

(0.56) 

(0.611) 

Assessor 4 

0.083 

0.21 

0.133 

0.433 

0.80 

0.619 


(0.196) 

(0.43) 

(0.421) 

(0.581) 

(1.22) 

(1.195) 

Assessor 5 

-0.101 

-0.17 

0.138 

-0.276 

-0.80 

0.002 


(0.200) 

(0.42) 

(0.440) 

(0.594) 

(1.20) 

(1.251) 

Assessor 6 

-0.066 

0.05 

0.135 

-0.935** 

-0.64 

-0.407 


(0.150) 

(0.31) 

(0.301) 

(0.445) 

(0.87) 

(0.855) 

Assessor 7 

-0.236 

-0.29 

-0.122 

-1.349*** 

-1.41 

-0.921 


(0.147) 

(0.33) 

(0.330) 

(0.438) 

(0.93) 

(0.937) 

R-squarcd 

0.794 

0.11 

0.186 

0.777 

0.12 

0.192 


Note: *** p<0.01, ** p<0.05, * p<0.1 N=66. Reference groups: OCS commissioning source; 
CRUDES ship types; assessor 8.CRUDES ships include destroyers, cruisers. Forward deployed 
ships are in Rota, Bahrain or Japan. West Coast ships include San Diego, CA, Everett, WA or 
Pearl Harbor, HI. Mean of pass competency is .576 and assessment score is 3.373. 
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2. Overall Underway and Simulator Experience Regression Analysis 

In Table 16 we analyze underway and simulator experience variables and their 
effect on both outcome variables. Due to the wide variation of experience amongst the 
officers, we chose the median values as our analysis point. We created binary variables that 
reflect whether an officer was above the median value for the experience variables listed 
in the table. Column 5 shows a correlation to OOD hours above the median and assessment 
score, as well as column 6 shows a positive correlation to underway OOD hours and 
assessment score. 


Table 16. Underway and simulator experience correlation to outcome 

variables 


Pass competency (score>=3) Assessment score (scale of 1 -7) 
(1) (2) (3) (4) (5) (6) (7) (8) 


Overall Experience 

Above median overall OOD hours, underway or simulator 0.007 0.590 

(0.141) (0.392) 

Above median overall OOD hours, underway -0.050 0.513 

(0.150) (0.422) 

Above median overall OOD hours, simulator -0.076 -0.139 


Above median overall watch hours, underway or simulator in any position 
Above median overall watch hours, underway in any position 
Above median overall watch hours, simulator in any position 


Covariates included 

yes 

Assessor FE 

yes 

R-squarcd 

0.112 


(0.135) (0.377) 



-0.017 




0.192 



(0.144) 

-0.090 



(0.409) 

0.224 



(0.145) 




(0.413) 



-0.026 




0.087 



(0.141) 




(0.402) 

yes 

yes 

yes 

yes 

yes 

yes 

yes 

yes 

yes 

yes 

yes 

yes 

yes 

yes 

0.119 

0.119 

0.112 

0.155 

0.147 

0.124 

0.122 


Note: *** p<0.01, ** p<0.05, * p<0.1 N=66. Covariates are amphibious ships, U.S. Naval 
Academy, NROTC, forward deployed ships and West Coast ships. Mean of pass competency 
is .576 and assessment score is 3.373. 


3. Overall Special Evolution Experience Regression Analysis 

Table 17 shows the regression analysis results of overall or total special evolutions 
experience. We again choose to analyze the variables based on the median data point. The 
evolution data is collected as each individual evolution category. For analysis, we also 
created a total sum variable of all evolutions conducted, which is reflected in Table 17. 
Column 2 reflects that the number of straits transits conducted is significant, and has a 
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positive correlation to passing the competency assessment. This result make sense because 
the OOD proficiency assessment is based around a straits transit. Therefore, it can be 
assumed the more experience an officer has with that type of evolution, then the better they 
would perform on the proficiency check. Additionally, Column 3 shows a significant 
relationship between the total number of all evolutions conducted and assessment score. 


Table 17. Special evolution experience correlation to outcome variables 



Pass competency 
(score>=3) 

(1) (2) 

Assessment score 
(scale of 1 -7) 

(3) (4) 

Overall Experience - Evolutions 





Above median overall number of evolutions conducted 

0.079 


0.708* 



(0.142) 


(0.392) 


Above median overall number of pier evolutions conducted 


-0.096 


0.120 



(0.164) 


(0.464) 

Above median overall number of unrep evolutions conducted 


0.070 


0.103 



(0.176) 


(0.500) 

Above median overall number of anchor evolutions conducted 


0.019 


-0.026 



(0.148) 


(0.419) 

Above median overall number of strait transit evolutions conducted 


0.357* 


0.235 



(0.181) 


(0.512) 

Above median overall number of high traffic evolutions conducted 


-0.120 


0.635 



(0.167) 


(0.475) 

Covariates included 

yes 

yes 

yes 

yes 

Assessor FE 

yes 

yes 

yes 

yes 

R-squared 

0.117 

0.198 

0.171 

0.208 


Note: *** p<0.01, ** p<0.05, * p<0.1 N=66. Covariates are amphibious ships, U.S. Naval 
Academy, NROTC, forward deployed ships and West Coast ships. Mean of pass competency 
is .576 and assessment score is 3.373. 


4. Recency of Underway or Simulator Experience Regression Analysis 

Table 18 presents our findings of recency of underway and simulator experience. 
Recency in our study is defined as having performed the watch in the past 90 days. We 
specifically created binary variables to describe recency data, with a one defining any hours 
and a zero meaning none. This binary indicator was chosen due to the low median recency 
values as shown in previous tables. Column 5 shows a correlation with overall OOD hours 
and assessment score, as well as OOD hours underway or in a simulator to assessment 
score. 
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Table 18. Recency of underway or simulator experience correlation to 

outcome variables 


Pass competency (score>=3) Assessment score (scale of 1 -7) 



(1) 

(1) 

(1) 

(1) 

(1) 

(1) 

(1) 

(1) 

Recency experience - in the past three months 









Any OOD hours, underway or simulator 

0.054 




0.584 




(0.154) 




(0.431) 




Any OOD hours, underway 


0.087 




0.407 





(0.161) 




(0.449) 



Any OOD hours, simulator 


-0.032 




0.535 





(0.173) 




(0.482) 



Any watch hours, underway or simulator in any position 



0.095 




0.610 





(0.158) 




(0.442) 


Any watch hours, underway in any position 




0.126 




0.511 





(0.165) 




(0.457) 

Any watch hours, simulator in any position 




-0.001 




0.524 





(0.169) 




(0.469) 

Covariates included 

yes 

yes 

yes 

yes 

yes 

yes 

yes 

yes 

Assessor FE 

yes 

yes 

yes 

yes 

yes 

yes 

yes 

yes 

R-squared 

0.114 

0.117 

0.122 

0.118 

0.149 

0.159 

0.167 

0.150 


Note: *** p<0.01, ** p<0.05, * p<0.1 N=66. Covariates are amphibious ships, U.S. Naval 
Academy, NROTC, forward deployed ships and West Coast ships. Mean of pass competency 
is .576 and assessment score is 3.373. 


5. Recency of Special Evolution Regression Analysis 

Table 19 shows our analysis of special evolutions recency to the two outcome 
variables. As we did with the Table 18, we created the binary variable indicting any special 
evolutions conducted for the same reason as before. We see in Column 3 a correlation of 
any evolutions being conducted in the past 90 days and assessment score. In Column 4, we 
see a varying effect of the individual special evolutions to assessment score, as well as their 
effect to passing the competency in Column 2. This may be due to the low number of 
observations, and the low recency numbers as illustrated in previous tables. 
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Table 19. Recency of special evolutions experience, correlation to outcome 

variables 


Pass competency Assessment score 
(score>=3) (scale of 1 -7) 

_ (1) (2) (3) (4) 

Recency experience - evolutions in the past three months 


Any evolutions conducted 

0.098 


0.604 



(0.153) 


(0.428) 


Any pier evolutions conducted 


0.172 


1.267** 



(0.182) 


(0.473) 

Any unrep evolutions conducted 


-0.094 


-1.068* 



(0.240) 


(0.622) 

Any anchor evolutions conducted 


-0.031 


-1.424** 



(0.246) 


(0.637) 

Any strait transit evolutions conducted 


-0.147 


-0.677 



(0.207) 


(0.536) 

Any high traffic evolutions conducted 


0.107 


1.228** 



(0.231) 


(0.598) 

Covariates included 

yes 

yes 

yes 

yes 

Assessor FE 

yes 

yes 

yes 

yes 

R-squared 

0.119 

0.139 

0.151 

0.288 


Note: *** p<0.01, ** p<0.05, * p<0.1 N=66. Covariates are amphibious ships, U.S. Naval 
Academy, NROTC, forward deployed ships and West Coast ships. Mean of pass competency 
is .576 and assessment score is 3.373. 
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V. ASSESSMENT RECOMMENDATIONS 


In this chapter, we will discuss our research recommendations regarding 
assessment improvements for the SWO community and SWOS. The development of a 
high-quality assessment system is a process that requires improvements over time. We 
recommend the following assessment and analysis improvements: 

1. Improve the assessment survey. 

2. Update assessment grade sheet to minimize subjectivity. 

3. Implement an electronic collection format for Surface Warfare Mariner 
Skills Logbook in addition to the current paper version. 

4. Continue OOD assessment research and expand its use to other SWO 
assessment timeframes. 

5. Increasing simulator training, specifically leverage simulators during in¬ 
port, maintenance and shipyard periods. 

A. IMPROVE ASSESSMENT SURVEY 

Currently, the OOD assessment survey format is conducted on paper, which 
requires students to handwrite in their responses. For NPS research, this requires SWOS to 
then scan each document and send PDF files for our conversion into an electronic format 
for software analysis, which greatly increases the possibility of measurement error. For 
example, in our research we removed three observations due to the low quality of data 
presented on the paper survey. Also, there are also issues ranging from legibility concerns 
to incorrect data input. We also noted that we need to think critically of the types of 
questions we are asking in the survey to ensure we are capturing the intended data. We 
have two recommendations to improve the survey data collection process of OOD 
assessments. 
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1. Implement Electronic Survey Format 

To improve the data collection process, we created an electronic format utilizing 
Microsoft Forms. Microsoft Forms allows for the storing of student responses in a cloud- 
based system, and a quick overview of received data and exportation to Microsoft Excel 
for analysis. A Microsoft Forms based survey significantly reduces the possibility of 
measurement error from student responses, as well as researcher data conversion from 
paper to an electronic format. Additionally, Forms allows restrictive question parameters 
to be set by researchers and SWOS. This setting also allows researchers to minimize 
inconsistent responses that may occur when students do not read questions carefully. Also, 
it allows the capability of requiring all questions be answered prior to submission. 
Microsoft Forms also expands the SWOS's capability of administering the survey to a 
potentially larger population, allowing more flexibility in future assessment periods. To 
utilize the survey, SWOS could email out the link to the intended recipients and track live 
responses, while electronically storing the data. For future research and analysis, we 
believe this is the next step in improving our assessment process, as well as improving our 
data storage. 

2. Improve Survey Questions 

During our research and in discussion with SWOS, we determined that the 
assessment survey may be collecting too much data, and we need to think critically about 
the questions presented. The change from the pilot study assessment survey (Appendix B) 
to the current assessment survey (Appendix C) was based upon Cunha and Dearth's (2019) 
research, and solved many issues discussed previously about the data format. Now that we 
are able to look at the new assessment methods, a more streamlined survey may result in 
better quality data. 

We recommend removing the questions about JOOD experience as we really care 
about CONN driving and OOD experience. Additionally, under special evolutions 
questions we should remove the high traffic density scenario, as it appears subjective, and 
this information is already specifically recorded as part of the Mariners Skill Logbook entry 
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(Appendix D). These changes result in a more streamlined survey that does not change the 
major key variables recorded, allowing year-to-year comparisons in future research. 

B. REMOVE SUBJECTIVITY FROM ASSESSMENT GRADE SHEET 

Previous improvements to the assessment (Cunha & Dearth, 2019) have been 
focused on removing subjectivity from the process and ensuring formality is followed. In 
January 2020, the Director of Fleet Training at SWOS, signed a memorandum to OOD 
participants (Appendix E). This memorandum formalizes the assessment process 
background, states the purpose and explains an overview of the process (Mahon, personal 
communication, 2020). Based on SWOS's intent and in further decreasing subjectivity, we 
recommend the following changes. 

The current assessment overall grading scale is a 1-7 scale where 1-unsatisfactorly, 
2-marginal, 3-below average, 4-average, 5-above average, 6-exceptional and 7-excellent. 
This scale provides significant variation but also instills a high amount of subjectivity. The 
difference between two grading points is vague and the grading sheet instructions provides 
minimal guidance to the graders. We recommend a 1-5 grading scale identical to the 
subcategory 1-5 scale. The overall assessment performance would then be 1-unsatisfactory, 
2-below average, 3- average, 4-above average, 5- excellent. 

The new scale will still allow significant variation for analysis but removes the 
subjectivity of the 1-7 scale. Additionally, we recommend adding general criteria of what 
each grade would mean while underway for performance guidelines. A grade of 1- 
unsatisfactory is a failure and would not result in qualification in the fleet. Grade 2-below 
average would result in a probationary type of qualification where a Captain may have the 
OOD stand daytime watches with an experienced JOOD and CONN to help when needed. 
Grade-3 is a trusted OOD that the Captain feels comfortable with them standing watch at 
night. Grades 4-above average and 5-excellent, are high performing OOD's who would be 
the officers a Captain would want training the more junior officers and future OOD's. 
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C. IMPLEMENT ELECTRONIC MARINER SKILLS LOGBOOK 

The Surface Warfare Mariner Skills Logbook brings a much-needed tracking of 
SWO experience, while also allowing future analysis based on these records. As the use of 
logbooks and the processes take hold in the Navy, we recommend research into an 
electronic logbook format that would be used in conjunction with the paper logbook. This 
is similar to how other entities within the military and in the civilian sector log their 
experience and track proficiency. Additionally, we recommend future research utilizing 
the end our tour summary letter data of the Mariner Skill logbook be utilized. Without 
research access to each individual's logbook, and the current lack of an electronic logbook 
format, only the end of tour summary letter provides researchers with the valuable 
information the logbooks contain. 

D. CONTINUE OOD ASSESSMENT PERFORMANCE RESEARCH 

We recommend continuing research, with a partnership with SWOS, to assist in 
answering the questions of what determines a high quality OOD and what the performance 
trend is year-to-year. Continuing the funding and research is critical for trend analysis and 
to determine effectiveness of the policy implementations by SWO community. The short- 
and long-term impacts of policy and program changes are only known if a quality 
assessment and feedback loop is implemented by an organization. This is what NPS and 
SWOS have intended to determine with the pilot study data and this current research. 

However, while the question may arise as to why the data from the pilot study 
wasn't combined with these efforts to improve statistical significance. The pilot study data, 
was important and was the initial attempt at answering performance questions. It ultimately 
looked at different population groups as well as lacked continuous data variable collection. 
With the changes that were made, our research was able to more accurately track OOD 
summary data and experience. While, the next phase of research will be able to utilize our 
data, due to minimal proposed future grading changes, and expand the number of 
observations to allow quality analysis results to be acquired. 

To allow analysis at the various phases of a SWO's career, we also recommend 
using this assessment grading criteria and survey in the other SWOS trainings and any 
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future fleet performance spot checks. By using similar grading criteria in all assessment, it 
allows researchers to properly assess each career milestone and creates the uniformity of 
expectations throughout the SWO community. Overall our proposal is to continue research, 
expand the number of observations to allow proper regression analysis and utilize the same 
assessment type in other SWO training schools overseen by SWOS. 

E. INCREASE SIMULATOR TRAINING 

Policy changes have increased the amount of formalized training a SWO receives 
throughout their career, but more specifically the amount of training received in their initial 
officer training tour. The data shows that the average time spent in the shipyard was about 
11 months out of a 24 month first-tour. 11 While almost an average of 50% of a tour was 
spent in the shipyard, the median 50% of officers conducted 10 hours of simulator training 
as an OOD and 40 hours in any position (OOD, JOOD or CONN). 

Our literature review research supports simulator training as being an effective and 
low risk training environment if implemented appropriately. As a result, we recommend 
researching the amount of time spent in simulators, specifically while in a maintenance 
availability or shipyard, and increase simulator requirements during these timeframes. 
Although maintenance is a priority during these timeframes, a high emphasis in the SWO 
community must be placed on professional development, training and maintaining 
proficiency during these periods. A SWO's core principal is safe operation of the ship, and 
the ability to conduct ourselves as professional mariners at all times. 

F. CONCLUSION 

The Surface Warfare Community and the Navy must continue to fund, support and 
improve assessment practices. A well-designed assessment is a feedback mechanism that 
allows analysis of implemented policies and training programs, while also providing 
insight into future policy shaping. The pilot study directly led into our research with the 
implemented grading, survey and data collection changes they proposed. We see a 


11 First-tour length has increased from 24 months to 30 months. Future research will reflect the longer 
first tour length and experience variables may or should increase due to this change. 
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significant improvement in the quality of data collected when compared to the pilot study 
but our proposed recommendations can further help to remove subjectivity that currently 
exists in the grading criteria. 

By working in a partnership with SWOS, our goal is to continue to improve the 
SWOS OOD assessment process. Our research recommendations are summarized as 1) 
move the data collection process forward to an electronic format 2) improve the grading 
criteria and remove subjectivity 3) continue the assessment research and expand to other 
assessment periods and 4) research simulator usage during in-port, maintenance and 
shipyard periods. 

Although we are not able to statistically determine the OOD proficiency 
determinants due to the low number of observations, continued research should be able to 
answer that question. In the future, the electronic collection methods will improve the data 
quality and allow timely analysis, while reducing measurement error. Additionally, 
logbook data can be utilized for increased accuracy and analysis. Ultimately, continued 
research will allow improved statistical and trend analysis of Surface Warfare Officer 
mariner skills proficiency, and support the Navy in training and maintaining our 
professional abilities to accomplish our mission. 
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APPENDIX A. LOGBOOK END OF TOUR SUMMARY LETTER 


Source: (COMNAVSURFOR, 2019) 


COMNAVSURFORINST 1412.7 
16 Dec 2019 

SAMPLE COMMANDING OFFICER END OF OFFICER’S TOUR LETTER TO 

PERS-41 

1412 

Ser CG XX/XXX 
DD Minin YY 

From: Commanding Officer, USS SHIP (CG XX) 

To: Commander, Navy Personnel Command (PERS-41) 

Subj: MARINER SKILLS LOGBOOK DATA SUMMARY OF ENS SALT E. SAILOR, USN, 
11XX 

1. During ENS Salt E. Sailor’s tour on USS SHIP (CG XX) from Month YYYY - Month 
YYYY. 

2. Watch Experience 

a. Underway Bridge Watch Hours 

(1) OOD 

(a) Qualified 

(b) Under Instruction 

(2) Bridge watch other than OOD or OOD (U/I) 

b. Bridge Watch Simulator Hours 

(1) OOD (qualified) or (under instruction) 

(2) Bridge watch other than OOD or OOD (U/I) 

(3) Dates attended BRM 

3. Special Evolutions Experience (Dav/Night) 

a. Sea and Anchor Details: 

b. Underway Replenishments: 

c. Anchoring Evolutions: 


59 





COMNAVSURFORINST 1412.7 
16 Dec 2019 


Subj: MARINER SKILLS LOGBOOK DATA SUMMARY OF HNS SALT E. SAILOR, USN, 
11XX 

d. Straits/Congested Waterway Transits: 

e. Other (specify particular evolution or event): 

4. CO’s Overall Assessment: (comments if desired). 


Is/ 

C. O. SHIP 


Copy to: 
ENS Sailor 
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APPENDIX B. PILOT STUDY ASSESSMENT SURVEY 


Pilot study research survey referenced in Cunha and Dearth's technical report. Source: (Cunha & 
Dearth, 2019). 


Name:_ Ship:_ 

Commissioning Source: 

O Naval Academy 

O ROTC: School:_ 

O OCS: 

Fill in the circle that closely applies: 

How many months have you been assigned to your current ship? 

0-6 7-12 13-18 >18 

O O O O 

Number of months you were on deployment? 

0-3 4-6 7-11 12-17 >18 

O O O O O 

Number of months it took you to qualify as an OOD? 

0-6 7-12 13-18 >18 

O O O O 

If SWO Qualified, the number of months it took you to qualify? 

Not Qualified 1-6 7-12 13-18 >18 

O O O O O 

I feel comfortable operating the RADAR/ARPA to its fullest extent in congested waterways? 

Yes No 

O O 

When using the VMS system on your ship, do you understand from where the system gets its inputs from and how the 
system determines your position? 

Yes No 

O O 

Do you feel confident when making Bridge to Bridge radio calls? 

Yes No 

O O 

When was the last formal BRM training you received from the Navigation, Seamanship, and Shiphandling Training 
(NSST) center? 

Never 1-3 months 4-11 months 12-18 month > 19 months 
O O O O O 


Watch station specific questions 

Conning Officer: The number of evolutions that you have conducted. 

Getting underway / Mooring to a pier: 


None 

1-2 

3-4 

5-6 

>6 

O 

O 

O 

O 

O 

Underway replenishment [approach): 



None 

1-2 

3-4 

5-6 

>6 

O 

O 

O 

O 

O 

Anchoring: 





None 

1-2 

3-4 

5-6 

>6 

O 

O 

O 

O 

O 

Transit a designated Traffic Separation Scheme (TSS): 


None 

1-2 

3-4 

5-6 

>6 

O 

O 

O 

O 

O 
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Number of straits transits conducted: 

None 1-4 5-8 9-12 >12 

O O O o O 

Approximate number of days underway where you were assigned Conning Officer on the watch bill: 
<20 21-99 100-200 >200 

O O O O 

As the Officer of the Deck (OOD): The number of evolutions that you have conducted. 

Getting underway / Mooring to a pier: 


None 

1-2 

3-4 

5-6 

>6 

O 

O 

O 

O 

O 

Underway replenishment (approach): 

None 1-2 3-4 

5-6 

>6 

O 

O 

O 

O 

O 

Anchoring: 

None 

1-2 

3-4 

5-6 

>6 

O 

O 

O 

O 

O 

Transit a designated Traffic Separation Scheme (TSS): 

None 1-2 3-4 5-6 

>6 

O 

O 

O 

O 

O 

Number of straits transits conducted: 



None 

1-4 

5-8 

9-12 

>12 

O 

O 

O 

O 

O 


Approximate number of days underway where you were assigned OOD on the watch bill: 
<20 21-99 100-200 >200 

O O O O 
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APPENDIX C. CURRENT OOD ASSESSMENT SURVEY 


Survey for this research created and provided through personal communication. Source: (Cunha & 
Dearth, personal communication, 2019). 


Officer nf the Deck Survey 

Name:_ Date:_ 

Home port:_ Age:_ Years of Service:_ 

Prior enlisted? Yes/No If yes, rating/MOS:_ Sex: Male / Female 

Commissioning Source: Naval Academy / ROTC / OCS / Maritime Academies 

If not the Academy, which school?_ 

Did you receive ship-handling training in school on a simulator? Yes/No 

Which ship did you serve on (e.g., USS Princeton CG-59)? _ 

When did you check onboard? (Month/year)_ 


When did you check-out of your first ship? (Month/year)_ 

When did you complete BDOC? (Month/year)_ 

Since you checked onboard, how long did you spend... 

In a shipyard or pier-side maintenance availability? _months 

In-port and not underway? _months 

Underway, not on deployment (including any cross-deck periods)? _months 

Underway, on deployment? (including any cross-deck periods)? _months 

*The total sum should equal the length of your first tour. _months 

When did you qualify OOD? (Month/year)_ 


When was your last formal BRM training with the Navigation, Seamanship, and Shiphandling 
Training (NSST) center? (Month/year)_ 

Are you SWO qualified? Yes/No If so, when did you qualify? (Month/year)_ 


Bridge Experience 

During normal underway steaming, were you in a three-person bridge team (OOD, JOOD, & CONN) 
or a four-person bridge team (OOD, OOD (U/I), JOOD, & CONN)? 3-person / 4-person 


Please provide the number of hours you have spent on the bridge and in simulators in various roles. 
(A 4 section watch for 6 months is approximately 1000 hours) 


As Conning Officer (CONN) 

As Junior Officer of the Deck 
(JOOD) 

As Officer of the Deck (OOD) 

As OOD Under Instruction (U/l) 


_ On the bridge _ _ In a simulator 

Hours in Hours in 

Total career last 3 Total career last 3 
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Special evolutions Please provide the number of evolutions that you have conducted on the bridge 
and in simulators in various roles. 

As a Conning Officer (CONN): 


_ In a simulator _ 

Total career Evolutions in 
evolutions last 3 months 


UNREPs 

Anchorings 

Straits or Traffic Separation 
Scheme transits 

High-traffic density watches 


_ On the bridge _ 

Total career Evolutions in 
evolutions last 3 months 

Sea & anchor details 
(Entering/exiting port) 


As a Junior Officer of the Deck (JOOD): 


Sea & anchor details 
(Entering/exiting port) 

UNREPs 

Anchorings 

Straits or Traffic Separation 
Scheme transits 

High-traffic density watches 


On the bridge 


Total career Evolutions in 
evolutions last 3 months 



In a simulator 


Total career Evolutions in 
evolutions last 3 months 



As an Officer of the Deck (OOD) or OOD (U/I): 


Sea & anchor details 
(Entering/exiting port) 

UNREPs 

Anchorings 

Straits or Traffic Separation 
Scheme transits 

High-traffic density watches 


On the bridge 


In a simulator 


Total career 
evolutions 


Evolutions in 
last 3 months 


Total career 
evolutions 


Evolutions in 
last 3 months 
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APPENDIX D. MARINER SKILL LOGBOOK EXAMPLE 


Example provided by a published instruction. Source: (COMNAVFURFOR, 2019, p. 2-1). 


Special Evolution Tracker 


Date 

Location: Coastal San Diego 

17Febl8 

Position 


Evolutions 

Traffic/Watch Complexity 

D 

N 

OOD 

CONN/JOOD 

OTHER 

S&A 

ANC 

RAS 

ST 

Other 

Low 

Med 

High 

X 


X 




X 




X 




Notes: 

- Daytime anchorage at Coronado Roads anchorage 118 

- Variable winds 

- .5 kts current, swells from the west 

- 75 yards from intended anchorage location 
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APPENDIX E. OOD COMPETENCY CHECK MEMORANDUM 


Memorandum provided by personal communication with SWOS. Source: (E. Cotter, personal 
communication, January 14, 2020). 


13 Jan 20 


MEMORANDUM 

From: Director, Fleet Training (N72), Surface Warfare Officers School Command 
To: Fleet OOD Competency Check Participants 

Subj: 2020 FLEET OOD COMPETENCY CHECK PROCEDURE 

Ref: (a) COMNAVSURFORINST 1412.7, SWO Career Manual 

1. Purpose . The Fleet Training Director (N72) Surface Warfare Officers School (SWOS) will 
conduct Officer of the Deck (OOD) Competency Checks from January through April in Fleet 
Concentration Areas. The candidates shall be randomly selected, 75% OOD qualified First Tour 
Division Officers on CRUDES, AMPHIB, and MCM ships. The remaining 25% will be Junior 
Officer of the Deck (JOOD) course graduates (started June 2019), with greater than 4 months on 
a ship since graduation, regardless of completion status of OOD qualification. 

2. Method . SWOS will send an assessment team to the Fleet Concentration Areas - Sasebo, 
Yokosuka, Mayport, San Diego, Norfolk, Everett, and Pearl Harbor. The checks will take place 
at local Navigation, Seamanship, and Shiphandling Trainers (NSST) or JOOD course facilities 
and assess both First Tour Fleet OOD and JOOD course graduate (“candidate”) abilities to, LAW 
the SWO Manual, Chapter 4, “effectively and safely handle and navigate a ship and lead a 
watchteam in a variety of light, medium, and high intensity virtual scenarios.” 

3. Candidate Requirements . Candidates are only required to bring their Mariner Skills Logbook 
as a reference for data collection in the experience survey. To maintain the integrity of the OOD 
Competency Check, candidates will review and sign a non-disclosure agreement and cannot 
discuss any portion of the assessment with other personnel. Uniform for all participants is NWU 
Type III. 

4. NSST/JOOD Facility Requirements . NSST/JOOD facilities will verify possession of the 
simulator scenario for required platforms and conduct preliminary runs to validate operation. 
NSST/JOOD system operators will provide overall control of the scenario and follow guidance 
from the assessor. 

5. Procedure . The OOD Competency Check has two parts, a NSS written test and a simulator 
scenario evaluation. Upon check-in, the candidates will complete experience surveys. For the 
simulator scenario, the candidate will fill the OOD role and SWOS personnel will fill the JOOD 
and CONN roles. The SWOS assessor will fill the Commanding Officer role. Prior to 
commencing the scenario, SWOS personnel will conduct a simulator familiarization and a scene- 
setter brief. After the scenario, the SWOS assessor will debrief the candidate. Results (No 
Concerns, Minor Concerns, Significant Concerns) will be forwarded to the candidate’s 
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Commanding Officer in a summary letter, including specific results from the written test and 
comments from the SWOS assessor. 


6. Schedule . TYCOM Force Navigators, via ISICs, will schedule candidates at each location. A 
notional daily schedule is provided below: 


Notional Daily Schedule 

Time 

Event 

0730 - 0800 

Morning group Check-in 

0800-0900 

Candidate written tests /1 st candidate in scenario 

0900- 1000 

2 nd candidate in scenario /1 st candidate written test 

1000- 1100 

3 rd candidate in scenario 

1100- 1200 

4 lh candidate in scenario 

1200- 1230 

Lunch 

1230- 1300 

Afternoon group Check-in 

1300- 1400 

Candidate written tests /1 st candidate in scenario 

1400- 1500 

2 nd candidate in scenario /1 st candidate written test 

1500- 1600 

3 rd candidate in scenario 


C. M. mXhon 
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